3D video applications in the consumer space have garnered great interest in the last few years since they are expected to further enhance a user's home multimedia experience but also since they are expected to revitalize certain applications such as High Definition DVD formats (i.e. Blu-ray). Nevertheless, several formats have been proposed for the delivery of 3D video into the home, separated primarily in three categories, frame compatible, 2D compatible, and 2D+depth formats. Each one of these categories has different benefits but also drawbacks and no clear winner in terms of the delivery format has been established.
Frame compatible formats rely on subsampling and multiplexing methods for the delivery of 3D data. More specifically, the left and right views are first subsampled to a lower resolution image given a particular filtering and subsampling mechanism. Common methods for subsampling include horizontal, vertical, and quincunx sampling. Multiplexing methods may include side by side (see FIG. 1(a)), over-under (see FIG. 1(b)), line interleaved, and checkerboard packing among others. In some cases, especially for side by side and over-under arrangements, one of the views may be inverted compared to the other view. These videos can be encoded using existing or potentially future codecs that better exploit the characteristics of the signal, while after the decoding process a device, i.e. a display, is provided with a single frame that now consists of the two sampled and multiplexed stereo images together. This device may have to demultiplex and then upsample the images for display or other purposes. The device may also have to reformat, i.e. convert from a format A such as side by side to a format B, such as line interleaved/line by line or checkerboard before display.
A major component of any video delivery system is the support of subtitles and graphic overlays. Existing systems, however, assume that the video is always in a 2D compatible format, i.e. that a single image corresponding to a scene covers the entire frame. See, for example, a conventional way of embedding subtitles in side by side and over-under packed video images, shown in FIG. 2(a) and FIG. 2(b). Unfortunately, frame compatible 3D formats complicate support for subtitles and graphic overlays. This also can affect some formats more than others. More specifically, formats such as “quincunx sampling+checkerboard interleaving”, “vertical sampling+line interleaving”, or “horizontal sampling+column interleaving” have less trouble in handling overlays and subtitles since these formats are pixel aligned with this additional information as embedded today. Unfortunately, this is not true for other formats, such as side by side (regardless of the sampling method), or over-under since the traditional way of embedding overlays in the video signal, i.e. across the entire frame, would be inappropriate and would result in a corruption of the overlay or subtitle text. This implies that one would have to either modify the decoder to support such overlays and render them differently for 3D applications. In an alternative method, and assuming that the graphic overlay or subtitles were added previously in a different part of the decoding & delivery chain, these have to be detected, extracted and re-embedded into the video, after also correcting, e.g. using inpainting or other techniques, any artifacts introduced from the original, not properly formatted and embedded overlays.